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Surveillance, method, system and module 

Tqqfrnical Field 

This invention relates to a method of monitoring a 
monitored location, a monitoring system and a monitoring 
module for monitoring. 
5 Background Art 

Monitoring of various public places , business pre- 
mises and private homes is becoming increasingly impor- 
tant as they contain increasingly valuable equipment, 
both of economic value, such as expensive computer equip- 
10 ment, and sentimental value, such as family heirlooms. 
In order to meet these increased monitoring needs, 
there are different types of monitoring systems. One type 
of monitoring system according to prior art normally com- 
prises a monitoring station and a number of monitoring 
15 modules, each of which monitors a monitored location . 

Each monitoring module is connected to a monitoring sta- 
tion via communication cables. Traditionally, a monitor- 
ing module is a video camera which continually records 
images which are transmitted to the monitoring station. 
20 In the monitoring station there are one or more alarm 
operators, who watch the transmitted images to decide 
whether any unauthorized activity is taking place, such 
as a burglar appearing at the monitored location. The 
disadvantage of this system is that the alarm operators 
25 must be paying attention continuously if there is anyone 
in the transmitted images. This severely limits the num- 
ber of monitoring modules that can be connected to the 
monitoring station, which also results in the monitoring 
being very expensive. In order to reduce the quantity of 
3 0 images transmitted to the monitoring station, the moni- 
toring module comprises a video camera and an infrared 
detector which are connected to each other. When the 
infrared detector detects a movement, a video recording 
is started. The video images can be transmitted via the 
3 5 communication cables to the monitoring station where an 
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operator views the images and makes a decision regarding 
measures to be taken, A problem with this type of moni- 
toring system is that in many cases the recorded images 
do not provide sufficient information about what has 
5 caused the alarm. This can occur when, for example, 
alarm situations, detected by the infrared detector, 
which have been caused by high temperatures or sabotage^ 
are not caught by the camera. In addition, the system 
still transmits a relatively large quantity of data. 

10 Patent application WO 98/28706 describes a monitor- 

ing system which comprises a number of cameras that are 
connected to a monitoring station. The cameras record 
images that are transmitted to the monitoring station. 
The monitoring station processes the images to determine 

15 whether there is an alarm situation or not. If it is 
determined that there is an alarm situation, an alarm 
signal is forwarded. 
Summary of th<* Invention 

An object of the invention is therefore to make pos- 

20 sible reliable and cost-effective monitoring and thereby 
to solve the above-mentioned problem. 

The monitoring system must also in a completely 
satisfactory way make it possible to protect the privacy 
of persons who are at the monitored location. 

25 The characteristics which belong to the extracted 

area can be calculated from data which represents the 
stylized area. 

These and other objects, which will become apparent 
from the following description, have now been achieved by 

30 a method of monitoring in accordance with claim 1. 

The invention is based on the knowledge of the 
advantages of working with objects which are extracted 
from an area. The area is a representation of an object 
which is detected at a monitored location. The object is 

35 created by producing some particular characteristics of 
the area, such as a stylized outline shape of the area. 
In other words, the object contains a reduced and limited 
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amount of information about the area, which information 
is sufficient to ensure whether there is an alarm situa- 
tion or not. By working with objects it is possible to 
create a first type of object which is used for classi- 
5 fication and a second type of object which is transmitted 
to a monitoring station for visual verification. These 
two types of object consist of the actual object or a 
subset of the actual object. In this way, classification 
of certain characteristics and a visual verification of 

10 other characteristics are made possible. 

According to one aspect, the invention thus com- 
prises a method of monitoring monitored locations by 
means of a monitoring system comprising a plurality of 
monitoring modules, each of which has a light-sensitive 

15 sensor, for monitoring the monitored locations, and a 
remote monitoring station with an operator, comprising 
the steps of recording by each of the monitoring modules 
an image of the monitored location associated with the 
monitoring module, extracting in each of the monitoring 

20 modules an area in the recorded image which differs from 
a reference image, extracting in each of the monitoring 
modules an object from the area, classifying in each of 
the monitoring modules the object , based on characteris- 
tics, such as a characteristic of the type: size, shape 

25 and/or movement history, associated with the object, if 
the object is a human alarm object, transmitting, if the 
object is classified as a human alarm object, data repre- 
senting the area in a stylized way to the monitoring sta- 
tion, and recreating said transmitted data in the moni- 

30 toring station and displaying the same to the operator 
for verification of the human alarm object. 

Thus, the invention comprises the step of recording 
images of a monitored location and of producing from 
these images information which is of interest for moni- 

35 toring purposes and transmitting certain information to. 
a monitoring station. 
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The monitored location is limited by the light- 
sensitive sensor and the field of vision of the associat- 
ed optics. The monitoring station is remote and can be a 
center belonging to a security company. Security person - 
5 nel is then sent to the monitored location immediately 
if an alarm is verified. The monitoring station can als X 6, 
for example, be connected directly to the police. 

The recorded image is compared with a reference 
image to detect new objects and events in the image. The 

10 reference image can be created by means of one or more 
algorithms from one or more previous images, one or more 
background images, or a combination of both. Averaging 
can be carried out of a number of said recorded images ^ 
to create a reference image. By means of the comparison, 

15 moving objects can be further processed and stationary 
objects, for example tables and chairs, that are in the 
monitored location can be excluded. This means that the 
areas contain interesting information about events in the 
monitored location. 

20 From the areas in the image which are different from 

the reference image, at least one characteristic is prc£- 
duced to create an object. The characteristic should be 
of such a type that it is of interest to study in the 
relevant monitoring situation. For example, if it is 

25 necessary to be able to distinguish between animals and 
people, certain specific characteristics, such as pat- 
terns of movement, can suitably be used. An object is 
created. A classification based on one or more character- 
istic is carried out to determine whether the object is 

30 a human alarm object. The monitoring system can be set up 
to classify the object as a human alarm object as soon as 
an object is determined to be a human object, but it can 
also be set up to classify the object as a human alarm 
object if the object is determined to be human and also 

35 fulfils some additional criterion, such as where in the 
image the human object is located. v . 
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The decision can be reached baaed on a comparison 
with predetermined threshold values and/or on charac- 
teristics of previously detected objects. 

The stylized depiction of the object that is trans- 
5 mi t ted when the object is classified as a human alarm 
object is suitable for narrow-band transmission while at 
the same time being able to be interpreted by the human 
eye for a verification that it is actually an alarm 
object. Data representing the stylized object comprises 

10 greatly data-reduced information about the extracted 

area. The data-reduced information still comprises suf- 
ficient information to enable it to be recreated in the 
monitoring station and displayed visually in such a way 
that an operator can verify reliably that there is 

15 actually an alarm situation at the monitored location. 
An operator can, for example, be a security guard or 
some other person who, in the event of an alarm being 
verified, contacts, for example, a security guard or the 
police. 

20 Data representing the area in a stylized way is 

transmitted via a communication medium, which can be, for 
example> a cable or a wireless connection, to a monitor- 
ing station. The object can be displayed to the operator 
on, for example, a display screen. The operator thus 

25 makes an evaluation of what he sees. If the operator, for 
example, judges that there is an intruder at the monitor- 
ed location, he takes suitable measures, such as sending 
security personnel to the location. On the other hand, if 
the operator judges that what he can see is not an alarm 

30 situation, no further action need to be taken and money 
is saved, as no unnecessary work needs to be carried out. 
In addition, the transmission medium is not overloaded 
with superfluous data, as the transmitted object com- 
prises a reduced and limited amount of information about 

35 the detected area. This means that a very large number 
of monitoring modules can be connected to one monitoring 
station. Analysis and dec is ion -making are distributed. 
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All computer-based analysis takes place in the monitoring 
modules and it can be sufficient to have only human veri- 
fication of the transmitted information in the monitoring 
station. In addition, with the limited amount of infor- 
5 mat ion it is easy to construct a queue system for the 
received alarm information in the monitoring station. 

The characteristics on which the classification is 
carried out can be calculated from data representing the 
stylized area. The advantage of this is that the operator 

10 verifies the decision whether there is an alarm situation 
or not using the same characteristics to make his deci- v 
sion as the decision engine in the monitoring module. 

In one embodiment, the step is included of creating 
the outline shape of the area in order to represent the 

15 area in a stylized way. 

The outline shape can be made up of a sequence of 
points in the edge of the area. Data about the size and 
shape of the area can be calculated relatively easily . 
from the outline shape. 

20 In one embodiment, the stylized area is a stylized 

outline shape. 

With the stylized outline shape, the quantity of 
data can be reduced as it does not comprise all the 
points in the edge of the area. Different algorithms can 

25 be used in order to produce the stylized outline shape. 
In the monitoring station there must be certain corre- * 
sponding algorithms so that the outline shape can be 
recreated and displayed visually. 

Data representing the stylized outline shape is 

30 transmitted. In the monitoring station the stylized out- 
line shape is recreated and displayed to the operator. 
The advantage of transmitting a set of data for the 
stylized outline shape is that it can be transmitted by 
narrow-band. The transmission can be carried out on a 

35 communication medium that has a bandwidth of less than 

10 kbit/s. In addition, the identity of the human-related 
alarm object is transmitted anonymously and is protected. 
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Problems can arise when a monitoring module sends an 
image of the monitored location to the monitoring sta- 
tion, as special permission is often required to use such 
a monitoring module in order to protect personal privacy. 
5 It can be difficult and complicated to obtain such per- 
mission. Monitoring modules in ordinary homes can also 
impose requirements associated with personal privacy. It 
is usually not desirable for people who live in the home 
which is monitored to be recorded on images, among other 

10 things as these images could be misused. 

The outline shape can be a good characteristic to 
make possible visual verification in the monitoring, sta- 
tion as to whether there is an alarm situation or not . 
The outline shape of a human-related object is 

15 relatively easy for an operator to recognize as a human 
figure, without the identity of the person being dis- 
closed. 

In another preferred embodiment, the step is includ- 
ed of comparing particular characteristics belonging to 

20 the object with corresponding characteristics belonging 
to ah object extracted out of a previously recorded 
image, in which case if the characteristics conform to 
the extent that they can be considered to belong to the 
same object, data is recorded about the associated move- 

25 ment history of the object for classification and/or 
transmission to the monitoring station to be recreated 
and displayed to the operator. 

If the compared characteristics conform to a certain 
predetermined extent, they are said to match and to ori- 

30 ginate from the same moving object, recorded at different 
times. For example, characteristics of the distinct 
region can be compared, such as its physical size in the 
image. For example, speed and direction of movement can 
be worked out. As the history of the object is known, it 

35 can be used as a characteristic on which to base deci- 
sions. The movement history can be displayed in the moni- 
toring station, together with the stylized outline shape, 
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as vectors which show the direction of movement and the 
speed. 

An advantage of displaying the movement history to 
the operator is that the decision regarding whether there 
5 is an alarm situation or not is made easier. The movement 
history which is displayed can also be an animation of 
outline shapes originating from objects extracted conse- 
cutively in time and representing the same extraneous 
object. The operator's evaluation of the alarm situation 

10 is made considerably easier when the pattern of movement 
associated with the object is displayed. For example, 
verification of persons becomes relatively simple, as 
they have a particular pattern of movement . An operator 
is able to analyze movement information which comes from 

15 a very large number of monitoring modules. k>. 
One embodiment further comprises, the steps of 
transmitting, if the object is classified as a human 
alarm object, supplementary alarm information about the 
area, such as information of the type: intensity regions 

20 and/or line content, together with data representing the 
area in a stylized way, and recreating and displaying the 
transmitted supplementary alarm information. 

The intensity regions make easier in particular the 
visual verification in the monitoring station, as the 

25 intensity regions make the display of the object clearer. 
This applies in particular when the object is human- 
related. For example, a darker lower part of the object 
displayed to the operator can represent trousers/skirt, 
which makes possible an easier interpretation. If it is 

30 the object's associated outline shape which is displayed, 
it can be filled in in a suitable way. 

Partial lines within the area are extracted. The 
line content gives the object more structure and provides 
essential information about the texture of the object. 

35 Examples of partial lines in a person can be that the 

chin is added so that the head is regarded as part of the 
rest of the body. 
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With a visual display in the monitoring station the 
partial lines make the decision- making concerning the 
alarm situation easier for the operator. It is easier to 
make out what the outline shape represents. 
5 According to a second aspect of the invention, this 

comprises a monitoring system for monitoring monitored 
locations, comprising a plurality of monitoring modules, 
each of which has a light-sensitive sensor for recording 
images of the monitored location, and a remote monitoring 

10 station, the monitoring modules being arranged to carry 
out computer-based analysis of the images, which com- 
prises extracting areas from the images which differ from 
a reference image, extracting an object from the area, 
classifying the object based on characteristics associat- 

15 ed with the object, such as a characteristic of the type: 
size, shape and/or movement history, and, if the object 
is classified as a human alarm object, transmitting data 
representing the area in a stylized way to the monitoring 
station, which is arranged to recreate and display said 

20 transmitted data to the operator for verification of the 
human alarm object. 

A further advantage of having analysis and decision- 
making in the monitoring module is that more reliable 
sabotage protection is achieved. If the monitoring is 

25 only carried out with a camera that forwards the image to 
a monitoring station, a burglar can, for example, cut the 
connection to the monitoring station, which means that 
no information about the burglary can be obtained. If a 
burglar sabotages the connection between the monitoring 

30 module and the monitoring station in a monitoring system 
according to the invention, the monitoring module conti- 
nues to record images and analyze these, and if there is 
an alarm situation the monitoring module can store the 
alarm object in its memory. In this way, the burglary can 

35 be recorded and information about the burglary can be 
retrieved from the monitoring module or sent when the 
connection is restored. The monitoring station can indi- 
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cate that the connection with a monitoring module is 
broken and an operator can go to the monitored location 
in question and investigate whether the broken connec- 
tion is due to sabotage. In addition, the operator can V. 
5 retrieve any stored alarm object from the monitoring 
module and in this way know whether there has been a 
burglary, for example. The monitoring module can also 
be arranged to store an image if the connection is broken 
and an alarm object is detected. The information which 

10 is stored requires little memory space, as not all the 

recorded images need to be stored, only data representing 
the alarm object, and possibly some individual images. * 
With traditional monitoring techniques it would not be 
possible to store the recorded images in the camera as 

15 this would be too memory- intensive. 

Further advantages of the monitoring system are 
apparent from the above discussion of the method. 

In one embodiment, the monitoring system comprises a 
central panel which is arranged to receive from at leas^t 

20 one subset of the monitoring modules said data represent- 
ing the area in a stylized way, and to forward this data 
to the monitoring station, together with supplementary 
data such as data of the type: date, time and informa- 
tion about from which monitoring module said data was 

25 received. 

The central panel can, for example, be located at 
an entrance to a building and also have the function 
of being able to activate and deactivate the monitoring 
modules. There can be a large number of central panels 

30 connected to the monitoring station. No processing of 

the received data is normally carried out in the central 
panel, except for the addition of extra information which 
can be of use to the operator in the monitoring station. 
In another embodiment according to the invention, 

35 the monitoring modules and the monitoring station are 
arranged to communicate by wireless means, such as by 
mobile telephony. 
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In wireless communication, the bandwidth for the 
transmission is particularly critical. With wireless 
communication no extra cable- laying is required, which 
reduces the costs. The monitoring station can, for exam- 
5 pie, be a mobile terminal. An advantage of this is that 
the operator does not need to remain in one particular 
place. The mobile terminal can, for example, be a mobile 
phone. As it is possible to show a simple object on a 
mobile phone's display, the operator can view the object 

10 on the mobile phone's display and can determine whether 
there is an alarm situation and based on this can take 
any measures. This means, for example, that the operator 
can carry out other tasks between alarm situations and 
when an alarm situation arises he is informed of this, 

15 for example, by means of an audio signal. 

The monitoring station can also be a server to make 
possible monitoring via a computer network. As no deci- 
sion needs to be taken by the monitoring station itself, 
this can be a server. An operator can monitor from any- 

20 where in the world, provided he has access to a network 
connection . 

According to a third aspect of the invention, this 
comprises a monitoring module for monitoring a monitored 
location, which comprises a light-sensitive sensor for 

25 recording an image of the monitored location, which moni- 
toring module is arranged to carry out computer-based 
analysis of the image, which comprises extracting an area 
from the image which differs from a reference image, 
extracting, an object from the area, classifying the 

30 object based on characteristics associated with the 

object, such as a characteristic of the type: size, shape 
and/or movement history, and, if the object is classified 
as a human alarm object, transmitting data representing 
the area in a stylized way to an external unit. 

35 With the monitoring module the same advantages are 

obtained as have been discussed above in connection with 
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the claims concerning the method and the claims concern- 
ing the system. In addition, the following is achieved.*' 

In a preferred embodiment, the memory is arranged 
to store a particular type of movement information for 
5 learning purposes. 

This has the great advantage that the monitoring 
module becomes better at ignoring false alarms and learns 
what does not give rise to an alarm situation. The saved 
movement information can, for example, be movement ;•. 
10 detected outside a window. Perhaps people often walk 
past, and are not objects that are to give rise to an 
alarm situation. This type of recurring movement in this 
area will then not give rise to an alarm situation. 

In one embodiment, a supplementary sensor is used 
15 which makes possible even more reliable monitoring. The 
accuracy of the whole system is thereby increased. The 
supplementary sensor can, for example, be an infrared 
detector. The infrared detector extends the monitored 
wavelength range. For example, it can be a useful addi- 
20 tion when an alarm object is wearing clothes which match 
the background in pattern and color, which can cause 
problems for the light-sensitive sensor. The infrared 
detector will then detect the object from the heat it 
is emitting. 

25 Brief Description of the Drawings U 
In the following, the invention will be described in 
greater detail utilizing an example of an embodiment and 
with reference to the accompanying schematic drawings, 
which illustrate a currently preferred embodiment of the 
30 monitoring system according to the invention. 

Fig. 1 shows a schematic diagram of the monitoring 
system according to one embodiment . 

Fig. 2 shows a schematic block diagram of the hard- 
ware in the monitoring module according to one embodi- 
35 ment. 

Fig. 3 shows a flow chart of a method of monitoring 
according to one embodiment. 
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Fig. 4 shows how the edge of an area is traced out 
according to one embodiment . 

Fig. 5 shows a line image in which all the edge 
points for the area are to be found according to one 
5 embodiment . 

Fig. 6 shows a polygonized image according to one 
embodiment . 

Fig. 7 shows a flow chart for matching an object 
according to one embodiment. 

10 Fig. 8 shows a general block diagram of an embodi- 

ment of the method of monitoring. 

Fig. 9a shows an example of how an alarm object can 
be displayed to an operator. 

Fig. 9b shows another example of how an alarm object 

15 can be displayed to the operator. 

Fig. 10 shows the monitoring system according to an 
embodiment of the invention. 
Description of a Preferred Embodiment 

Fig. 1 shows schematically a monitoring system with 

20 a number of monitoring modules 1 which can communicate 
with a monitoring station 3 via a transmission medium 2. 

Fig. 2 shows a block diagram of the hardware in the 
monitoring module 1. The monitoring module 1 is supplied 
with a voltage to a voltage connection 4. In addition, 

25 the monitoring module 1 co^ri£^s._a.jaowsrfial calgula ting 
unit 5. The monitoring module 1 comprises a communication 
unit 6. In addition, the monitoring module 1 comprises a 
light-sensitive sensor „7„^_fpr..exair©lfi_a_£MOS sensor, for 
recording imaas^. The sensor 7 is integrated into a chip 

30 and also has a lens arrangement 8. The sensor 7 provides 
an analogue output signal which is forwarded to an A/D 
converter 9 for conversion into a digital signal. In 
addition, the monitoring module 1 comprises, a random 
^ccM£jnemory 10. The monitoring module 1 operates ^with 

35 a suitable operating system and can carry out advanced 
image processing. The monitoring module l also comprises 
a permanent memory 11 for computer code and other data 
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which has to be saved in a non- volatile memory. In addi- 
tion, a lighting device 12 can be arranged in association 
with the monitoring module 1 to illuminate dark monitored 
locations. The lighting can advantageously be carried out 
5 in the infrared range as the monitoring module 1 will 
then not emit any visible light, which will make it very 
difficult to find in dark monitored locations. This 
results in increased reliability, as the danger of sabo- 
tage is reduced. Infrared diodes are also cheap and do 

10 not use much power. The monitored location is limited by 
the field of vision of the sensor 7 and its associated 
optics. All the components comprised in the monitoring 
module 1 are advantageously integrated on a circuit 
board. The advantage of this is that the monitoring 

15 module 1 is much more stable, that is it is much less 
sensitive to interference and has fewer points where 
sabotage can occur. 

The alarm criteria of the monitoring module 1 are 
stored in the permanent memory 11 and can be changed from 

20 the monitoring station 3 by the transmission of new soft- 
ware from the monitoring station 3 to the monitoring 
module l. The alarm criteria can be different for diffe- 
rent monitoring modules 1. The alarm criteria can be 
changed dependent upon the permitted power consumption 

25 and external conditions. The external conditions can, for 
example, be a monitor that is switched on, curtains that 
move or other permitted movements that take place at the 
monitored location. The alarm criteria are, of course, 
also set in accordance with the type of object and/ or 

30 pattern of movement for which the monitoring module 1 is 
to give an alarm. 

With reference to Fig. 3 and Fig. 9, the monitoring 
function of the monitoring module 1 will now be describ- 
ed. In a recording step 100, the sensor 7 continually 

35 records imaaes of the man i hnrpH legation. A recorded 

image is converted in a conversion step 110 from an ana- 
log signal to a digital signal in the A/D converter 9. In 
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the calculating unit 5 a difference image is created in a 
difference step 115 by a subtraction between a reference 
1 imaae and the recorded image in question. The reference 

image can be created using one or more algorithms from 
5 one or more previous images, one or more background 
images or a combination of both. Averaging or Kalman 
filtering can be carried out on a number of said recorded 
images to create a reference image. The reference image 
is normally updated at regular intervals. Areas are 

10 extracted from the difference image in an area extraction 
step 120, for example by refining the difference image, 
and we get what we call a divergence image which is shown 
in Fig 8 . The resulting areas consist of a defined num- 
ber of pixels in the recorded image. Several areas are 

15 mutually exclusive so that a particular pixel can only 

belong to one area. Areas represent a change of some kind 
that has occurred in the recorded image in cbmparrsSn 
with the, .reference imass *- -.These changes ~carrr*T^^^ple, 
be that a person has entered the monitored location, a 

20 , feirdlhas flown into the monitored~T6caEidiI Or ~a TreeT is 
_ b^owinc^ in the wind in the monitored location. In a fil- 
tration step 130 a conventional imageTT l It rar X6n can be 
carried out to remove noise. When the areas have been 
extracted, an object, is associated with each area in an 

25 object extraction step 140 for easier management of the 
different areas. Instead of storing images of an area, 
selected area characteristics are stored, such as one or 
more of, for example, coordinates in the image, size, 
outline, average intensity, circumference and intensity 

30 variations. 

With reference to Figs 4-6, a method will now be 
described of producing the outline of the area, which 
area in this case represents a person. Fig. 4 shows how 
an extracted area is traced out along its edge by a 

35 search function which has a clock-hand algorithm. The 
clock-hand algorithm traces along the edge of the area 
until it reaches the point where it started. In detail, 
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the following takes place. A starting point is first 
looked for on the edge of the area. As long as no starfe* 
node is encountered and there are unexplored ways for- 
ward, a clock-hand is moved clockwise at a distance of 
5 one pixel from the previous position until a new edge 
point is encountered. If the clock-hand's new position 
is the start position then a new unexplored way is 
looked for. If there is no way forward, the algorithm is 
to be discontinued. Otherwise the algorithm continues and 

10 the unexplored way forward from the start node which was 
found is marked as explored. 

Fig. 5 shows an outline of an area which represents 
a person. In Fig. 6 a polygon has been fitted to the 
traced-out path. The polygon is adjusted using an angle - 

15 minimizing function. The angle-minimizing function is as 
follows. A starting point is set on the edge as the most 
recent point. As long as the end point is ^ 
not encountered, then the edge is traced out. The angle 
difference between the tangent vector of the most recent 

20 point and the tangent vector of the present position 
around the edge is calculated. If the angle difference 
is greater than a particular limit, then this position 
is saved as a node, and the position is set as the most 
recent point. It is also possible to use other types of 

25 outline shapes than polygons, for example, splines. A 
spline curve is defined mathematically by a number of 
control points and a function which describes the appear- 
ance of the curve between the control points. Normally, 
the function is fixed and only the control points are 

30 used to define the curve. To fit such a curve to an out- 
line image it is necessary to have an initial value, a 
criterion for where the curve fits the outline and a 
search strategy to fit the curve to the outline. Normal- 
ly, the position of the curve in the previous image in a 

35 sequence of images is used as an initial value. If start- 
ing from scratch, another method must be used, for exam- 
ple by starting with a large circle that is guaranteed to 
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include the outline. The criterion for fitting the curve 
to the outline can be either the distance to the detected 
outline or based on the gradient in the image. In the 
latter case, it is required that the curve should be 
5 placed where the gradient is the greatest. The search 
strategy consists normally of some standard optimiza- 
tion method in order to minimize the criterion in the 
search strategy. The advantage of a spline representa- 
tion for optimization is that only the control points 

10 need to be used as variables, which leads to increased 
speed. For more details about spline fitting, see the 
article "Fast least-square curve fitting using quasi - 
orthogonal splines", Myron Flickner, James Hafner, 
Eduardo J. Rodriguez and L. C. Sanz. 

15 After combining the object's characteristics the 

object is stored in a storage step 150 in the form of 
its characteristics in the random access memory 10 and 
a matching of the object with a stored object from a 
previously recorded image is carried out in a matching 

20 step 160. The objects' characteristics are compared with 
each other to produce an indication of how well they 
conform. By minimizing the matching difference for all 
objects at the same time, a good approximation is obtain- 
ed of the object's previous history, which is known as 

25 tracking. Matching is carried out by gradual stages in 
such a way that it is only the object in the most recent 
image that is compared with what was stored from the pre- 
vious image or, alternatively, from earlier images. After 
matching it can be seen, for a particular object whether 

30 the object was recorded in a previous image and if so 
which object it was in the preceding image. As the pre- 
vious object possibly in turn has a direct connection 
to the previous object, a chain is built up of the total 
history of the current object. 

35 The matching method is illustrated in Fig. 7 and 

comprises the following. The object is compared with all 
previous objects which were extracted from the previous 
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image in a combining step 200. A calculation of the 
degree of matching of the combinations is carried out in 
a calculation step 210. The outcome of the calculation of 
the matching is normalized so that the result is a value 
5 between 0 and 1. The value 0 indicates that the compared 
objects do not have any characteristics that conform, 
while the value 1 indicates that the objects are precise- 
ly identical. If the combination with the highest degree 
of matching, for the object and a previous object, 

10 exceeds a predetermined value, it is. determined in a 
determination Btep 220 that there is a match. In the 
matching step 160 a decision is taken concerning a number 
of characteristics of the object, which characteristics 
are also weighted according to their importance. A method 

15 that increases the probability of a correct matching is 
to extract the intensity regions within an object. The 
method is based on segmenting an area based on- one of its 
intensity characteristics. The different segments have an 
average intensity and a specified area. Different methods 

20 can be used for the segmentation itself. Examples of such 
methods are quantifying of the intensities, refining of 
the intensities or classification of different pattern 
segments by means of, for example, Bayer's classifica- 
tion about which more can be read in R.C. Gonzales, 

25 R.E. Woods, "Digital Image Processing", Addison Wesley. 
The different segments can then be saved efficiently 
in various ways. One way is to save the outlines of the 
different segments or carry out a Run Length Encoding 
(RLE) of the different segments as a pixel map. 

30. In the classification step 170 the object is classi- 

fied based on the object's history and characteristics, 
based on which the decision engine can determine whether 
the object is an alarm object or not. For each recorded 
image the decision engine has access to all the objects 

35 and their histories which were extracted from the image. 
Initially, the decision engine views the whole history of 
the different objects and determines from this whether it 
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is a human alarm object. It is sufficient for an object 
to have been an alarm object at some time during its 
history for this to give rise to an alarm situation for 
the rest of its life. The object must fulfil a number of 
5 criteria in order to be classified or a decision reached 
concerning its being an alarm object. In order to achieve 
a particular level of confidence, for example, its his- 
tory must be sufficient. For example, it can be determin- 
ed that in order to cause an alarm a particular object 

10 must have been followed for at least 10 images back in 
time. Other criteria for classifying the object as an 
alarm object can be that for the whole of its life it 
covered a certain minimum distance and had a top speed 
which is not less than a certain lower limit. 

15 If the object is classified as an alarm object, that 

is it is classified as a person whose behavior is not 
permitted, data representing the area in a stylized way 
is forwarded in a transmission step 180 via a communica- 
tion cable 2 to an operator at a monitoring station 3 for 

20 display of the object in a display step 190. The trans- 
mission can be carried out at less than 10 kbit/s and 
still transmit a sufficiently large amount of information 
to make possible a verification of the alarm object. How 
many of the alarm object's characteristics are sent to 

25 the operator, and when, can be varied and determined by 
the users of the monitoring system. 

If there are several objects in a recorded image, 
they are all investigated in the same way. 

What- is transmitted and displayed to the operator in 

30 the display step 190 is the outline of the area. The out- 
line can be displayed as an animation corresponding to 
the recorded object in recordings made consecutively in 
time. Fig. 9a shows one alternative for displaying the 
object to the operator. This shows the object's present 

35 outline and a series of previous outlines which show how 
the object has moved based on previously recorded images. 
Figs 9b shows a further alternative for the display to 
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the operator. The outline of the object shows where the 
object has been situated in different recordings. 

In addition, data which represents the line content 
of an object can be sent together with the outline shape. 
5 The main aim of visualizing the line content in the area 
is to give the visual display of the transmitted informa- 
tion of the object more structure and essential informa- 
tion about the nature of its texture. There are a number 
of different sets of lines that can be extracted from a 

10 texture. Edges can be refined out of the derived texture. 
The whole area of the object can be made thinner and iris 
this way a kind of "stickman" is obtained. This stickman 
is quite sensitive to local changes and is therefore not 
always suitable. In addition, it originates from the out- 

15 line and not from the texture. The texture can be regard- 
ed as a topography. A set of lines can be all the hill- 
tops that can be described purely mathematically as, for 
example, saddle points and local maximums and minimums, 
etc. The lines are usually not particularly thin, but * 

20 often have some form of width. In order to obtain nar- 
row distinct lines, a method can be used that is called 
"thinning". Thinning "eats away" the edges of the thick 
lines without them being "eaten away" completely. 
Expressed simply, all the lines are made equally narrow 

25 (usually 1 pixel in width) . In certain cases, the result 
is not a number of individual lines, but more of a gric^. 
Then all the partial lines can be regarded as separate 
lines and separated from the other lines. In order to 
make the visual result as clear as possible, it can some- 

30 times be necessary to weed out the information. For exam- 
ple, if there is a checked shirt in the texture, there 
can be quite a lot of lines clustered together. The 
weaker lines or some of those which are too close toge- 
ther can then advantageously be removed. Finally, the K 

35 lines can be represented in a number of different ways. 
One way is in the form of pixels. Each line is described 
by the set of pixels it contains. Another way is line 
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sequences. A line sequence is fitted to each line seg- 
ment- Each line is represented here by a series of 
straight lines which together approximate to the original 
line. A further way is in the form of a spline. A spline 
5 is fitted to the line in question. 

In addition, intensity regions can be sent with both 
the outline shape and the line content or only with the 
outline shape in order to make easier a visual evaluation 
which, for example, takes place in this case when the 

10 outline shape is displayed to the operator. The intensity 
regions are to reproduce as closely as possible the char- 
acteristic features of an object. In order to achieve a 
good segmentation it is first necessary to define which 
characteristics of the texture of the object, belong toge- 

15 ther. Examples of such characteristics can be that the 
whole area is to have the same intensity with only small 
deviations. Another characteristic can be that the 
variance of the area is to be less than a particular mea- 
surement. A further characteristic can be that the area 

20 has a particular set of statistical characteristics such 
as average value, variance, correlation between adjacent 
pixels, etc. There are different ways of segmenting the 
different areas. In order to segment the different areas 
with the characteristics as mentioned above, a number of 

25 different methods can be used. One way is "Split and 

Merge" which is an algorithm that successively divides an 
area into smaller areas until the various partial areas 
fulfil a particular requirement. Subsequently the areas 
which have the same characteristics are combined. 

30 Another way can be quantifying the area at a low 

bit-depth to give distinct regions. A further way is to 
plant a seed in the texture and to let this area grow as 
long as the new pixel conforms with the characteristics 
of the new area. Pixels are marked as allocated when they 

35 are included in an area. When an area cannot grow any 
larger, then this area is completed and a new seed is 
planted in another location. It is also possible to have 
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a plurality of seeds growing at the same time in paral- 
lel. Another way can be Bayes classification according to 
a number of selected region characteristics in the tex- 
ture . 

5 In order to represent the different regions, a num- 

ber of different methods can be used. A first method is 
"Run Length Encoding" (RLE) of the di-fferent regions' * 
pixels. The value of the different pixels is which area 
they belong to. Another method is polygon representation. 

10 This method fits a polygon to the area. The polygon can 
share points with other areas and with the outline of the 
object. A further method is spline representation which 
delimits the area by a spline. An advantage is that the 
amount of data is smaller and the fit is better. A dis- 

15 advantage is, however, that most spline methods cannot 
share common points and that the fitting is more calcu- 
lating-intensive. 

Once the regions and the lines have been represent- 
ed, it is only a set of data that is sent via a transmis- 

20 sion medium. The only restriction is that both the trans- 
mitter and the receiver, which in this case are the moni- 
toring module 1 and the monitoring station 3, must inter- 
pret the information in the same way. They must have the 
same model of the information. 

25 With reference to Fig. 10, a further embodiment 

according to the invention will now be described. A num- 
ber of monitoring modules 20 are arranged at suitable 
locations in a building. These monitoring modules 20 are 
connected by wireless means to a cent ral= panel 21 which 

30 is arranged at the entrance to the building. By means of 
the central panel 21 the alarm in the building can be 
activated and de-activated. The central panel 21 is in 
turn in wireless communication with a monitoring station 
22. The monitoring station 22 has a number of central 

35 panels 21 connected to it. Each monitoring module 20 con- 
tinually records images of the location it is monitoring. 
An image recorded in a monitoring module 20 is compared 
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in the monitoring module 20 with a reference image and 
any divergence areas are extracted. When a divergence 
area has been extracted, an object is derived with asso- 
ciated characteristics such as the object's size, shape, 
5 direction of movement and speed. The direction of move- 
ment and the speed can be derived according to the above- 
mentioned technology- The outline shape of the object is 
also derived according to the above-mentioned technology. 
The object is classified in the monitoring module 20 

10 based on its characteristics. If the person moves in a 
particular direction at a particular speed, the person 
is said to constitute an alarm object. If the object is 
classified as an alarm object, data about the outline 
shape and data about the object's direction of movement 

15 and speed are transmitted to the central panel 21. The 
central panel 21 is arranged to add information about 
the date, time and in which monitoring module the alarm 
occurred. The data now contains the outline shape , direc- 
tion of movement, speed of movement, date, time and 

20 information about which monitoring module 20 gave the 
alarm. No processing of the recorded alarm object is 
carried out in the central panel 21. The said data is now 
forwarded to the monitoring station 22. The monitoring 
station 22 comprises monitors which are monitored by 

25 alarm operators. The received outline shape is shown on 
the monitors and also the direction and speed of the 
object by means of arrows. The date, time and from which 
monitoring module 20 the alarm came are also shown on the 
monitors. The alarm operator can now reach a decision 

3 0 regarding appropriate further measures. If a monitoring 
module 20 is put out of action for any reason, for exam- 
ple by sabotage, the central panel 21 sends information 
about this to the monitoring station 22, which can then 
put the fault right. 

35 Even though a special embodiment of the invention 

has been described above, it will be obvious to a person 
skilled in the art that many alternatives, modifications 
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and variations are possible in the light of the above 
description. Communication can take place via radio, for 
example GSM or Bluetooth. Instead of the outline shape, 
for example, other types of stylized information about * A 
5 the object can be transmitted which make it possible for 
the operator to see what the transmitted data represent- 
ing the object represents. For example, the skeleton form 
of the object can be transmitted or some type of informa- 
tion where the object is filled in so that its shape can 
10 be displayed visually. 
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CLAIMS 

1. A method of monitoring monitored locations by 
5 means of a monitoring system comprising a plurality of 

monitoring modules (1), each of which has a Jight-sensi- 
tive sensor, for monitoring the monitored locations, and 
a'remote monitoring station (3) with an operator, com- 
prising the steps of 
10 recording by each of the monitoring modules (1) an 

image (100) of the monitored location associated with the 
monitoring module, 

extracting (120) in each of the monitoring modules 
an area in the recorded image that differs from a refe- 
l^fe rence image, 

extracting in each of the monitoring modules an 
object (140) from the area, 

Qlassifvinq in each of the monitoring modules (1) 
the object based on characteristics, such as a charac- 
20 teristic of the type: size, shape and/or movement his- 
tory, associated with the object, if the object is a 
human alarm object, 

transmitting, if the object is classified as a human 
alarm object, data representing the area in a stylized 
25 way to the monitoring station (3) , and 

recreating said transmitted data in the monitoring 
station (3) and displaying the same to the operator for 
verification of the human alarm object. 

2. A method according to claim 1, in which the 

30 method further comprises the step of creating the out- 
line shape of the area in order to represent the object 
in a stylized way. 

3. A method according to any one of claims 1 or 2, 
in which the stylized area is a stylized outline shape. 

35 4. A method according to any one of the preceding 

claims, further conprising the step of comparing parti- 
cular characteristics associated with the object with 
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corresponding characteristics, associated with an objectV 
extracted from a previously recorded image, in which case 
if the characteristics conform to the extent that they 
can be considered to belong to the same object, data is 
5 recorded about the movement history of the object for 
classification and/or transmission to the monitoring 
station to be recreated and displayed to the operator. 

5. A method according to any one of the preceding , 
claims, further comprising the steps, if the object is 

10 classified as a human alarm object, of transmitting sup- 
plementary alarm information about the area such as 
information of the type: intensity regions and/or line 
content together with data representing the area in a 
stylized way and of recreating and displaying the trans - 

15 mi t ted supplementary alarm information. 

€. A monitoring system for monitoring monitored ^, 
locations, comprising a plurality of monitoring modules 
(1), each of which has a light-sensitive sensor for 
recording images of the monitored locations, and a remote 

20 monitoring station, the monitoring modules being arranged 
to carry out computer-based analysis of the images, which 
comprises extracting areas from the images that differ 
from a reference image, extracting an object from the 
area, classifying the object based on characteristics, 

25 such as a characteristic of the type: size, shape and/or 
movement history, associated with the object, and if the 
object is classified as a human alarm object, transmit- 
ting data representing the area in a stylized way to the 
monitoring station (3) which is arranged to recreate and 

30 display said transmitted data to the operator for verifi- 
cation of the human alarm object. 

7. A monitoring system according to claim 6, further 
comprising a central panel which is arranged to receive 
said data representing the area in a stylized way from at 

35 least a subset of the monitoring modules, and to forward 
this data together with supplementary data, such as data 
of the type: date, time and information about from which 
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monitoring module said data was received, to the monitor- 
ing station. 

8. A monitoring system according to claim 6 or 7, in 
which the monitoring modules and the monitoring station 

5 are arranged to communicate by wireless means, such as by 
mobile telephony. 

9. A monitoring module (1) for monitoring a moni- 
tored location, which comprises? a memory, a light-sensi- 
tive sensor for recordincr an image of the monitored loca- 

10 tion, and a communication device for communication with 
an external unit and a calculating unit for detecting a 
moving object from the recorded information, which moni- 
toring module is arranged to carry out computer-based 
analysis of. the image, which comprises extract inq an 

15 area from the imaae that differs from a refe3cen£e image, 
extracting an object from the area, classifying the 
object based on characteristics, such as a characteristic 
of the type: size, shape and/or movement history, 
associated with the object, and, if the object is clas- 

20 sified as a human alarm object, transmitting data repre- 
senting the area in a stylized way to an external unit. 

10. A monitoring module (1) according to claim 9, in 
which the memory is arranged to store a particular type 
of movement information for learning purposes. 

25 11- A monitoring module (1) according to claim 9 or 

10, in which the monitoring module (1) comprises a sup- 
plementary sensor. 



WO 01/48719 



1/10 



PCT/SE00/02665 




Fig. 1 



SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



PCT/SEOO/02665 



2/10 




SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



PCT/SE00/02665 



3/10 



Record image 



i 



Conversion to 
a digital signal 



Create divergence 
image 



Area extraction 



Image filtering 



Object extraction 



Storage of object 



Object matching 



ALARM? 



■100 

110 
115 

120 

130 
140 

150 
160 



NO 



170 



u YES 



Transmission of object 



180 



Display of object to 
alarm operator 



190 



Fig. 3 



SUBSTITUTE SHEET (RULE 26) 




SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



PCT/SE00/026&5 



5/10 




Fig. 5 



SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



PCT/SE00/02665 



6/10 




Fig. 6 



SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



7/10 



PCT/SEO0/02665 



200 ^ 



Combine objects 

i — ■ 



210 



Calculate 
degree of matching 



220 \, 



Is a predetermined threshold 


NO 


value exceeded? 


► 




YES 











No matching 



There is matching 



Fig. 7 



SUBSTITUTE SHEET (RULE 26) 



WO 01748719 



PCT/SEOO/02665 



8/10 




SUBSTITUTE SHEET (RULE 26) 



WO 01748719 



PCT/SE00/02W5 



9/10 

— r 




Fig. 9b 



SUBSTITUTE SHEET (RULE 26) 



WO 01/48719 



PCT/SE00/02665 



10/10 




SUBSTITUTE SHEET (RULE 26) 



V. 



INTERNATIONAL SEARCH REPORT 



Interna ti on aJ application No. 

PCT/SE 00/02665 



A. CLASSIFICATION OF SUBJECT MATHER 
IPC7: 608B 13/196 

According to ImgnaljonaJ Patent Classification (IPC) or to both natjonaj daaaificaUon and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (daswficatim system followed by classification symbols) r 

IPC7: G08B 



Documentation searched other than minimum documentation Ui the extent that such document* are included in the Deals searched 

SE,0K,FI,K0 classes as above 



Electronic data base consulted during the international search {name of data base and, where practicable, cearch terms used) 



C DOCUMENTS CONSIDERED TO BE RELEVANT 



Category* 


Glation of document, with indication, where appropriate, of die relevant passages 


Relevant to claim Vlo. 


A 


IEEE Transactions on circuits and systems for video 
technology, Vol 9 t Mr 7 October 1999, Foresti G. 
"Object Recognition and Tracking for Remote Video 
Surveillance", p, 1045-1062, see chapter II, 
System Description, fig.l 


1-11 


A 


Meyer M.et al. n A hew system for video-based 

detection of moving objects and its integration 
into digital networks."In: Security 
Technology, 1996. 30th annual 1996 International 
Carnahan Conference, p. 105-110, see chapter II, 
Description of the objekt detection concept, 
figures 1-3 


1-11 



J?J Further documents are listed in Oie continuation of Box C [j^ Sec patent family annex. 



* Special categories of died « 

A - " document deDring the general Arte of the art which is not a roddercd 
to be of particular relevance 

* E* carter ^ajplj cation or patent but published cm or after the intemational 

*L" document which may throw doubts on priority datsf*) or which is 
cited to establish the publication dale of another dlation or other 
special reaxon (aa apedfled) 

"O" document referring to an oral disclosure, uac, exhibition or other 



P" document published prior to the rmcmaUanal filing dale but later than 
the priority date d aimed 



* [** later document puhliahed after the mtcmaUorui fifing dale or priority 
date and not in conflict with the application but died to uaderstand 
the principle or theory under! ying the invention 

*X a document of particular relevance: the domed invention cannot be 
considered novel or cannot he cunsccred to involve an inventive 
step when the document is taken alone 

"Y" document or particular relevance: the d timed invention cannot he 
considered to involve an inventive step when the document is 
combined with one or more other such docavtcnU, such combin alien 
being obvious to a person stilled fat the art 

"cV document member of the lame patent family 



Date of the actual completion of the international search 
29 March 2001 


Date of mailing of the mlernaUonal search report 

0<2 -04- 2001 


Name and mailing address of the ISA/ 
Swedish Patent Office 
Box 5055, S-1Q2 42 STOCKHOLM 
Paaimile No. +46 8 666 02 86 


Authorized officer 

GSran Magnus son /it* 

Telephone No. +46 ft 782 25 00 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/SE 00/02665 



C (Continuation). DOCUMENTS CONSIDERED TO Bis RELEVANT 



Category* 



Citation of document, whh indication, where appropriate, of the relevant passages 



Relevant to daim No. 



Stringa,E.et al. "Content-based retrieval and real 
time detection from video sequences aquired by 
surveillance systems. "In -.Image Processing, 1998. 
ICIP 98. Proceedings. 1998 international Conference, 
Vol. 3, p. 138-142. see chapter 2.Asurvei11ance 
system for detecting lost objects, figures 2,3 



WO 9828706 AI (ESCO ELECTRONICS CORPORATION), 
2 July 1998 (02.07.98), figures 1,6, 
abstract 



EP 0805426 A2 (KELLY S. ET AL), 5 November 1997 
(05.11.97), claim 2 



1-11 



1-11 



1-11 



Form PCT7ISA/2JO {continuation ofxecomi sheet) (July 1998) 



INTERNATIONAL SEARCH REPORT 

Information on patent family members 



25/02/01 



International application No. 

PCT/SE 00/02665 



Patent document 
died in jcarch report 



Publication 
dale 



Patent family 
n)embcr(s) 



Publication 
dale 



M0 



9828706 A 



02/07/98 



AU 
EP 
US 
US 



5810998 A 
1010130 A 
5937092 A 
5956424 A 



17/07/98 
21/06/00 
10/08/99 
21/09/99 



EP 



0805426 A2 05/11/97 



GB" 9609406 0 



00/00/00 



Form PCT/lSA/ll 0 (patent family annex) (July IWJ) 



